3.5. Recurrent Neural Network (RNN) In Practice - LSTM Text Generation

This article continues from 3.3. Recurrent Neural Network (RNN) Theory (Advanced). If you have not read that one yet, it is recommended that you start there.

3.5.1. Preparation

Please run the following command in your terminal to install the required packages:

pip install keras tensorflow scikit-learn

This article uses my txt file, which I uploaded to GitHub. Click the link to view and download it. We use Mary Shelley’s Frankenstein as the training data.

Just place the downloaded data in your Python project.

3.5.2. Model Architecture

In this article we will use a Long Short-Term Memory (LSTM) model for text generation. We will feed the model the dictionary obtained after converting the text, then let it predict the 21st character from the first 20 characters and calculate accuracy.

Each input sample is a sequence of 20 time steps (characters). The LSTM layer has 20 units, and the number of neurons in the output layer depends on how many different characters appear in the text.

3.5.3. Coding It Up

Step 1: Load the Data

First, we load the data and remove line breaks at the same time:

# Load the data  
data = open('Frankenstein_by_Mary_Shelley.txt').read()  
  
# Remove line breaks  
data = data.replace('\n', ' ').replace('\r', '')

Step 2: Build the Dictionary

Each character in the text needs its own entry in the dictionary. First we remove duplicate characters and then build the dictionary:

# Remove duplicate characters  
letters = list(set(data))  
  
# Print for checking  
print(len(letters))  
  
# Build the dictionary  
char_to_int = {b:a for a, b in enumerate(letters)}

• By using set(data), we remove duplicates from all characters in the text and keep one unique copy of each character, such as letters, numbers, punctuation marks, and so on. list(set(data)) turns those unique characters into a list and stores them in the variable letters.

• The dictionary char_to_int uses the corresponding character as the key and an integer starting from 0 as the value, so that characters can be converted to numeric data and then to one-hot format.

Output:

74

These 74 different characters (uppercase and lowercase letters, digits, punctuation, and so on) make up the entire text.

Step 3: Input Processing

Next we need to process the string in batches. Here we first define a data_processing method, and later we can call it to extract the X and y data the model can receive:

# Batch processing of characters  
import numpy as np  
from tensorflow.keras.utils import to_categorical  
  
# Implement batch processing for character-based text input: the full text, sliding-window size, and conversion dictionary  
def data_processing(data, slide, num_letters, char_to_int):  
    # Extract data with a sliding window  
    def extract_data(data, slide):  
        x = []  
        y = []  
        for i in range(len(data) - slide):  
            x.append([a for a in data[i:i + slide]])  
            y.append(data[i + slide])  
        return x, y  
  
    # Convert characters to integers in batches  
    def char_to_int_data(x, y, char_to_int):  
        x_to_int = []  
        y_to_int = []  
        for i in range(len(x)):  
            x_to_int.append([char_to_int[char] for char in x[i]])  
            y_to_int.append([char_to_int[char] for char in y[i]])  
        return x_to_int, y_to_int  
  
    char_data = extract_data(data, slide)  
    int_data = char_to_int_data(char_data[0], char_data[1], char_to_int)  
    Input = int_data[0]  
    Output = list(np.array(int_data[1]).flatten())  
    Input_Reshaped = np.array(Input).reshape(len(Input), slide)  
    new_ = np.zeros((Input_Reshaped.shape[0], Input_Reshaped.shape[1], num_letters))  
    for i in range(Input_Reshaped.shape[0]):  
        for j in range(Input_Reshaped.shape[1]):  
            new_[i, j, :] = to_categorical(Input_Reshaped[i, j], num_letters)  
    return new_, Output

extract_data function:

• Purpose: extract samples from the input character sequence using a sliding window.
  • x: the list of characters inside the window (length = window size).
  • y: the target character immediately following the window (the prediction target).
• Logic:
  • x is obtained with [a for a in data[i:i + slide]], which takes the characters within the current window range.
  • y takes the first character after the window, data[i + slide].
  • Example: if slide=3 and data="abcdefg", the result is:
    • x = [['a', 'b', 'c'], ['b', 'c', 'd'], ['c', 'd', 'e'], ['d', 'e', 'f']]
    • y = ['d', 'e', 'f', 'g']

char_to_int_data function:

• Purpose: convert the character sequences x and target characters y into integer sequences.
  • It uses the char_to_int dictionary to get the integer index of each character.
  • Example: if the input is x = [['a', 'b', 'c'], ['b', 'c', 'd']], y = ['d', 'e'], and the dictionary is {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4}, the result is:
    • x_to_int = [[0, 1, 2], [1, 2, 3]]
    • y_to_int = [[3], [4]] (later flattened to [3, 4] in Output)

Conversion to model input:

• Through extract_data and char_to_int_data, we extract the data and convert characters to integers;
• The original integer input Input has shape [number of samples, window size];
• Output is the target character sequence, stored as a one-dimensional array.

One-hot encoding processing:

• Purpose: convert the integer input Input_Reshaped into a 3D one-hot input new_.
  • Input_Reshaped[i, j] is the integer value of the jth character in the ith sample of the window.
  • to_categorical(Input_Reshaped[i, j], num_letters) builds the one-hot encoding of the character with dimension [num_letters].
  • The encoding result of each sample is stored in new_, whose shape is [number of samples, window size, number of characters].

Final output:

• new_: one-hot encoded input array with shape [number of samples, window size, vocabulary size], suitable for deep-learning models such as LSTMs.
• Output: integer array of target sequences, one-dimensional, with length equal to the number of samples.

Step 4: Get the Model Input

We now use the data_processing method to extract the X and y data the model can receive, while also confirming that the time step is 20 (the model predicts the next character based on the previous 20 characters):

# Time step  
time_step = 20  
  
# Get the model input data  
X, y = data_processing(data, time_step, len(letters), char_to_int)

# Check the data structure  
print(X.shape)  
print(len(y))

Output:

(421397, 20, 74)
421397

• (421397, 20, 74):

  • 421397: the number of samples extracted by the sliding window, that is, how many [input sequence -> target character] samples were built. This depends on the relationship between the text length len(data) and the sliding-window size time_step.
  • 20: the number of time steps per sample, i.e. the sliding-window size time_step, which means each input sequence contains 20 characters. This value is determined by time_step = 20 in the code.
  • 74: the number of character categories, i.e. the one-hot encoding dimension. This comes from the vocabulary size len(letters), meaning there are 74 unique characters in the input data, including letters, punctuation, spaces, and so on.

• y is the list of target-character integers for the samples. Each sample has one target character, so the length of y is the same as the number of samples, namely 421397.

Step 5: Split the Data

Split the data into training and test sets:

# Split the data  
from sklearn.model_selection import train_test_split  
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=10)  
  
# Convert y_train to one-hot encoding  
y_train_category = to_categorical(y_train, num_classes=len(letters))

• The main reason for converting y_train into one-hot encoding is so it can be used to train the LSTM model

Step 6: Build the Model

The model takes sequences of 20 time steps; the LSTM has 20 units, and the number of neurons in the output layer depends on the number of different characters in the text, that is, the length of the dictionary:

# Build the model  
from tensorflow.keras.models import Sequential  
from tensorflow.keras.layers import Dense, LSTM  
  
model = Sequential()  
  
model.add(LSTM(units=20, input_shape=(time_step, len(letters)), activation='relu'))  
model.add(Dense(len(letters), activation='softmax'))

Step 7: Configure the Model

# Configure the model  
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Step 8: Train the Model

Just feed the training data to the model:

# Train the model  
model.fit(X_train, y_train_category, epochs=20, batch_size=1000)

batch_size=1000:

• The number of samples used in each training step (gradient update).
• The data will be split into multiple batches, each containing 1000 samples.
  • For example, if there are 50,000 training samples and each batch is 1,000, then 50 updates are needed in one epoch.
  • Of course, if you have a lot of memory, you do not need to set this parameter and can load everything into memory at once, although it would take a terrifying 15 GB or so, not even counting the IDE.

Step 9: Evaluate the Model

We evaluate the model using accuracy:

# Training-set accuracy  
_, train_accuracy = model.evaluate(X_train, y_train_category)  
print(f"Train Accuracy: {train_accuracy}")  
  
# Test-set accuracy  
y_test_category = to_categorical(y_test, num_classes=len(letters))  
_, test_accuracy = model.evaluate(X_test, y_test_category)  
print(f"Test Accuracy: {test_accuracy}")

Output:

Train Accuracy: 0.4766451120376587
Test Accuracy: 0.4727812111377716

Even though this accuracy is still below 50%, keep in mind that the computer has to guess one answer out of 74 different characters, and this result is already quite good. Random guessing would give only 1/74, or 1.35%, so the model’s accuracy is more than 30 times higher.

You can try to improve accuracy further by using a larger LSTM, stacking layers, training longer, or sampling from the predicted distribution to generate longer text.